Arm Plugin

The robotic arm model, found in the gazebo-arm.world file, calls upon a gazebo plugin called the ArmPlugin. This plugin is responsible for creating the DQN agent and training it to learn to touch the prop.
The gazebo plugin shared object file, libgazeboArmPlugin.so, attached to the robot model in gazebo-arm.world, is responsible for integrating the simulation environment with the RL agent. The plugin is defined in the ArmPlugin.cpp file, also located in the gazebo folder.

The ArmPlugin.cpp file takes advantage of the C++ API covered earlier. This plugin creates specific constructor and member functions for the class ArmPlugin defined in ArmPlugin.h. Some of the important methods, that you will require for the project, are discussed below:

ArmPlugin::Load()

This function is responsible for creating and initializing nodes that subscribe to two specific topics - one for the camera, and one for the contact sensor for the object.

In gazebo, subscribing to a topic has the following structure:

gazebo::transport::SubscriberPtr sub = node->Subscribe("topic_name", callback_function, class_instance);

Where,

callback_function is the method that’s called when a new message is received, and

class_instance is the instance used when a new message is received.

You can refer to the documentation for more details on the above -

For each of the two subscribers, there is a callback function defined in the file and is discussed below.

### ArmPlugin::onCameraMsg()

This is the callback function for the camera subscriber. It takes the message from the camera topic, extracts the image, and saves it. This is then passed to the DQN.

ArmPlugin::onCollisionMsg()

This is the callback function for the object’s contact sensor. This function is used to test whether the contact sensor, called my_contact, defined for the object in gazebo-arm.world, observes a collision with another element/model or not.

Furthermore, this callback function can also be used to define a reward function based on whether there has been a collision or not.

ArmPlugin::createAgent()

Previously, for the fruit and catch samples, you created a DQN agent. The createAgent() class function serves the same purpose wherein you can create and initialize the agent. In ArmPlugin.cpp, the various parameters that are passed to the Create() function for the agent are defined at the top of the file, some of them are:

#define ALLOW_RANDOM true
#define DEBUG_DQN false
#define INPUT_WIDTH   512
#define INPUT_HEIGHT  512
#define INPUT_CHANNELS 3
#define OPTIMIZER "None"
#define LEARNING_RATE 0.0f
#define BATCH_SIZE 8
#define EPS_START 0.9f
#define EPS_END 0.05f
#define EPS_DECAY 200
#define GAMMA 0.9f

ArmPlugin::updateAgent()

For every frame that the camera receives, the agent needs to take an appropriate action.

Since the DQN agent is discrete, the network selects one output for every frame. This output (action value) can then be mapped to a specific action - controlling the arm joints. The updateAgent() method, receives the action value from the DQN, and decides to take that action.

There are two possible ways to control the arm joints:

Velocity Control
Position Control

For both of these types of control, you can increase or decrease either the joint velocity or the joint position, by a small delta value. You can experiment with several different approaches to define on what basis the joint velocity or position changes. For example, one such action can be based on whether the action value is odd or even.

ArmPlugin.cpp includes the variable VELOCITY_CONTROL, set to false by default, which can be used to define whether you want to control joint velocities or positions.

ArmPlugin::OnUpdate()

This method is primarily utilized to issue rewards and train the DQN. It is called upon at every simulation iteration and can be used to update the robot joints, issue end of episode (EOE) rewards, or issue interim rewards based on the desired goal.

At EOE, various parameters for the API and the plugin are reset, and the current accuracy of the agent performing the appropriate task is displayed on the terminal.